Orientation determination utilizing a cordless device

ABSTRACT

A system for generating orientation information includes a reflector, an image collection system, and a processor. The image collection system is configured to collect at least two sets of image data, where one set of image data includes a stronger indication of the reflector than the other set of image data. The two sets of image data are collected using at least one reflector with some orientation-specific characteristic or a reflector integrated into a device that includes some orientation-specific structural feature. Once collected, the two sets of image data are used to generate orientation information related to the reflector. In particular, orientation information related to the reflector is generated by taking the difference between the two sets of image data.

BACKGROUND OF THE INVENTION

Most position tracking systems used with a graphical user interface(GUI) utilize a mouse to generate two-dimensional position information.The mouse is typically tethered to the computer by an electrical cordthrough which power is provided from the computer to the mouse andposition information is provided from the mouse to the computer. Acordless mouse utilizes a rechargeable or replaceable battery as itspower source and radio frequency (RF) signals to communicate positioninformation to the computer. While conventional position trackingsystems work well, the electrical cord of a corded mouse can restrict auser's freedom of movement and the power source of a cordless mouserequires constant recharging or replacement.

In addition to the above-described limitations, a conventional mouseonly provides two-dimensional position information. Three-dimensionalposition information and orientation information are desirable in someposition tracking applications.

SUMMARY OF THE INVENTION

A system for generating orientation information includes a reflector, animage collection system, and a processor. The image collection system isconfigured to collect at least two sets of image data, where one set ofimage data includes a stronger indication of the reflector than theother set of image data. The two sets of image data are collected usingat least one reflector with some orientation-specific characteristic ora reflector integrated into a device that includes someorientation-specific structural feature. Once collected, the two sets ofimage data are used to generate orientation information related to thereflector. In particular, orientation information related to thereflector is generated by taking the difference between the two sets ofimage data. Because one of the sets of image data includes a strongerindication of the reflector than the other set of image data, thedifference image gives a definitive indication of the reflector andbecause the image data is captured using a reflector with someorientation-specific characteristic or a reflector integrated into adevice that includes some orientation-specific structural feature, thedifference image includes orientation-specific image information relatedto the orientation of the reflector.

The use of a reflector and an image collection system enablesorientation information to be generated using a passive pointing devicethat does not require any power. Therefore, the pointing device does notneed to be tethered to a computer system or equipped with a battery. Theposition of the pointing device can be tracked while the pointing deviceis moved in free space.

Other aspects and advantages of the present invention will becomeapparent from the following detailed description, taken in conjunctionwith the accompanying drawings, illustrated by way of example of theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a system for generating position information thatincludes a retroreflector, an image collection system, and a processor.

FIGS. 2 and 3 illustrate the detection angle, the on-axis illuminationangle, and the off-axis illumination angle in the image collectionsystem of FIG. 1.

FIG. 4 depicts light of wavelength λ1 and wavelength λ2 that is incidenton an imager after reflecting off of a retroreflector and anymiscellaneous objects.

FIG. 5 illustrates the basic operation of the system of FIG. 1.

FIG. 6 is an exemplary graph of difference image data with the intensitydifference values on the x-axis and the number of data points collectedat each intensity difference value on the y-axis.

FIG. 7 depicts an alternative embodiment of the image collection systemfrom FIG. 1 that includes one imager and multiple on-axis and off-axislight sources.

FIG. 8 depicts an embodiment of an image collection system that includestwo imagers and a single light source.

FIG. 9 depicts an embodiment of the image collection system thatincludes two imagers and light sources that provide light of twodifferent wavelengths.

FIG. 10 is a perspective view of the image collection system of FIG. 9configured in an epipolar geometry.

FIG. 11 depicts an embodiment of an image collection system thatincludes on-axis and off-axis light sources and a wavelength-selectivefilter formed over the retroreflector.

FIG. 12 depicts an exemplary embodiment of an image collection system inwhich the light sources of two different wavelengths have the same, orroughly the same, illumination angle and in which a wavelength-selectivefilter is formed over the retroreflector.

FIG. 13 depicts an embodiment of an image collection system thatincludes one imager and a single light source.

FIG. 14 depicts another embodiment of a single light source imagecollection system.

FIG. 15 depicts an embodiment of a system for generating positioninformation that utilizes a reflector other than a retroreflector.

FIGS. 16A and 16B depict an embodiment of a pointing device thatincorporates a mechanism for shielding and revealing a retroreflector.

FIGS. 17A and 17B depict another embodiment of a pointing device thatincorporates a mechanism for shielding and revealing multipleretroreflectors.

FIGS. 18A and 18B depict an embodiment of a pointing device thatincludes a retroreflector embedded into a cavity of a pointing device.

FIG. 19A depicts a top view of an imager that includes a checkerboardfilter pattern.

FIG. 19B depicts a side view of the imager from FIG. 19A with a hybridfilter over the imager.

FIG. 20 depicts a portable computer configured with an image collectionsystem that includes two light sources and one imager embedded into thehousing of the computer.

FIG. 21 depicts a portable computer configured with an image collectionsystem that includes two light sources and two imagers embedded into thehousing of the computer.

FIG. 22A a front view of an image collection system in which an imagerand light sources are incorporated into a free standing housing.

FIG. 22B depicts a side view of the image collection system of FIG. 22A.

FIG. 23 depicts a process flow diagram of a method for generatingposition information.

FIG. 24 depicts a process flow diagram of another method for generatingposition information.

FIGS. 25A and 25B depict an example of a retroreflector that has anorientation-specific shape.

FIGS. 26A and 26B depict an example of multiple retroreflectorsconfigured in an orientation-specific pattern on the surface of apointing device.

FIGS. 27A and 27B depict an example of multiple retroreflectors thathave different shapes.

FIGS. 28A-28C depict an example of a retroreflector that is integratedinto a device that has an orientation-specific structural feature.

FIGS. 29A-29C depict an embodiment of a pointing device that is similarto the pointing device described with reference to FIGS. 28A-28C exceptthat it includes a retroreflector that is not subject to theorientation-specific structural feature of the pointing device.

FIGS. 30A and 30B depict an embodiment of a pointing device in whichmultiple retroreflectors are used in conjunction withorientation-specific structural features to enable the generation ofmore detailed orientation information.

FIG. 31A depicts a side view of the pointing device from FIGS. 30A and30B in which the cavities and retroreflectors are oriented relative to abeam of light such that all of the retroreflectors are illuminated.

FIG. 31B is a front view of the pointing device from FIG. 31A in whichthe illumination of all of the retroreflectors is represented by thefilled in dark circles.

FIG. 32A depicts a side view of the pointing device from FIGS. 30A and30B in which the cavities and retroreflectors are oriented relative tothe beam of light such that only some of the retroreflectors areilluminated.

FIG. 32B is a front view of the pointing device from FIG. 32A in whichthe illuminated retroreflectors are represented by filled in darkcircles and the non-illuminated retroreflectors are represented by theun-filled in circles.

FIGS. 33A and 33B depict an example of a pointing device that utilizesan orientation-specific pattern of retroreflectors as described withreference to FIGS. 26A and 26B in conjunction with a pattern ofretroreflectors that is integrated into orientation-specific features ofthe pointing device as described with reference to FIGS. 30A-32B.

FIG. 34 depicts an embodiment of a cavity in a pointing device and aretroreflector that is attached at the bottom of the cavity.

FIG. 35 depicts a process flow diagram of a method for generatingorientation information.

FIG. 36 depicts a process flow diagram of a method for generatingposition information and orientation information.

Throughout the description similar reference numbers may be used toidentify similar elements.

DETAILED DESCRIPTION

A system for generating orientation information includes a reflector, animage collection system, and a processor. The image collection system isconfigured to collect at least two sets of image data, where one set ofimage data includes a stronger indication of the reflector than theother set of image data. The two sets of image data are collected usingat least one reflector with an orientation-specific characteristic or areflector integrated into a device that includes an orientation-specificstructural feature. Once collected, the two sets of image data are usedto generate orientation information related to the reflector. Inparticular, orientation information related to the reflector isgenerated by taking the difference between the two sets of image data.Because one of the sets of image data includes a stronger indication ofthe reflector than the other set of image data, the difference imagegives a definitive indication of the reflector and because the imagedata is captured using a reflector with some orientation-specificcharacteristic or a reflector integrated into a device that includessome orientation-specific structural feature, the difference imageincludes orientation-specific image information related to theorientation of the reflector.

FIG. 1 depicts a system 100 for generating position information thatincludes a retroreflector 102, an image collection system 104A, and aprocessor 106. In the embodiment of FIG. 1, the image collection systemincludes an imager 108 and first and second light sources 110, 112 thatoutput light 114, 116 having different wavelengths. The first lightsource 110 is located closer to the imager than the second light source112 and the two light sources and the imager are located in the sameplane. The first light source is referred to herein as the “on-axis”light source and the second light source is referred to herein as the“off-axis” light source.

The terms “on-axis” and “off-axis” as used herein are described withreference to FIGS. 2 and 3. The angle at which light from a light sourceilluminates the retroreflector 102 is referred to as the illuminationangle. Referring to FIG. 2, the illumination angle is measured relativeto a plane (as indicated by dashed line 120) that is coplanar with theplane of the major surface of the imager 108. The illumination angle oflight source 110 is identified as θ_(i1) and the illumination angle oflight source 112 is identified as θ_(i2). The angle at which light 118reflected from the retroreflector is incident on the imager is referredto as the detection angle. The detection angle is identified as θ_(d)and is measured relative to the same plane as the illumination angle. Inthe case of two illumination angles, θ_(i1) and θ_(i2), the term“on-axis” applies to the light source or the illuminating beam whoseillumination angle has the smaller difference from the detection angle,θ_(d), and the term “off-axis” refers to the light source or theillumination beam whose illumination angle has the largest differencefrom the detection angle. Referring to FIG. 2, the relationship of theillumination and detection angles is mathematically expressed as:|θ_(i1)−θ_(d)|<θ_(i2)−θ_(d)|.

FIG. 3 illustrates changes in the illumination and detection angles thatoccur as the retroreflector 102 moves relative to the imager 108 and thetwo light sources 110, 112. Although the illumination and detectionangles change as the retroreflector moves, the difference between theon-axis illumination angle and the detection angle remains smaller thanthe difference between the off-axis illumination angle and the detectionangle, |θ_(i1)−θ_(d)|<|θ_(i2)−θ_(d)|. Given this relationship betweenthe illumination and detection angles, the status of a light source asan on-axis light source or an off-axis light source does not change withmovement of the retroreflector. It should be noted that thisrelationship is still maintained when the retroreflector is moved in adirection having a component perpendicular to the plane of the figure.

Referring back to FIG. 1, the image collection system 104A is used tocollect at least two sets of image data. The first set of image data iscollected in response to light at wavelength λ1 generated by lightsource 110 and the second set of image data is collected in response tolight of wavelength λ2 generated by light source 112. The first set ofimage data, referred to herein as the on-axis image data, isrepresentative of light of wavelength λ1 that reflects off theretroreflector 102 and any other objects that are illuminated by lightsource 110. The second set of image data, referred to herein as theoff-axis image data, is representative of light of wavelength λ2 thatreflects off the retroreflector and any other objects that areilluminated by light source 112. Because the on-axis illumination angleis closer to the detection angle than the off-axis illumination angle,the on-axis image data will include a stronger indication of theretroreflector relative to the off-axis image data. In this case, theintensity of light reflected by the retroreflector to the imager 108will be much greater at wavelength λ1 than at wavelength λ2. Therefore,the intensity values at data points related to the retroreflector willbe higher in the on-axis image data than at corresponding data points inthe off-axis image data. FIG. 4 depicts light 122, 124, 126 ofwavelength λ1 and wavelength λ2 that is incident on the imager afterreflecting off the retroreflector and other objects 128. While theintensity of light 126 reflected by the retroreflector to the imagerwill be much greater at wavelength λ1 than at wavelength λ2, theintensity of light 122, 124 reflected by the other objects will beroughly the same assuming that the intensity of the light emitted by thetwo light sources is roughly equal.

The difference in the intensities of light that is reflected from theretroreflector 102 at the two wavelengths is used to generate positioninformation. The basic operation of the system of FIG. 1 is describedwith reference to FIG. 5. In FIG. 5, a retroreflector 102 is connectedto a pointing device 130 that is held in a hand 132. The hand, pointingdevice, and retroreflector are illuminated by light sources 110 and 112from FIG. 1. On-axis and off-axis image data 136, 138 are collected bythe imager while the objects are illuminated by the respective lightsources. As illustrated in FIG. 5, the on-axis image data 136 includeslight that reflects off the hand, the pointing device, and theretroreflector. Because the illumination angle at wavelength λ1 and thedetection angle are similar, the on-axis image data includes a strongindication of light from the retroreflector. The strong indication oflight is represented in FIG. 5 by a dark spot. The off-axis image data138 also includes light that reflects off the hand, the pointing device,and the retroreflector. However, because the off-axis image data isobtained in response to off-axis light, the intensity of the off-axislight that reflects off the retroreflector towards the imager 108 issmall compared to the intensity of the on-axis light that reflects offthe retroreflector towards the imager. Therefore, in the off-axis imagedata, the retroreflector does not appear significantly brighter than anyof the other objects.

In comparing the on-axis image data 136 to the off-axis image data 138,the only significant difference between the two data sets (assuming thedata sets are captured simultaneously or nearly simultaneously) is theindication of the retroreflector 102. In view of this, the position ofthe retroreflector can be definitively identified by taking thedifference between the two sets of image data to produce a differenceimage 140. Because most of the data points in the two sets of image dataare the same, most of the corresponding data points will cancel eachother out, with the one exception being the data points that correspondto the retroreflector. Therefore, the difference between the on-axisimage data and off-axis image data gives a definitive indication of theretroreflector's position. As depicted in FIG. 5, the difference imagedefinitively identifies the retroreflector while excluding any of theother objects. Referring back to FIG. 1, the difference operation iscarried out by the processor 106.

Because a difference function is relied upon to generate positioninformation, it is important to generate two sets of image data that canbe efficiently compared. In one embodiment, light detected by the imageris filtered to limit the detected light to wavelength bands around thewavelengths used for illumination. For example, when light ofwavelengths λ1 and λ2 is used for illumination, a bulk filter withtransmission peaks at wavelengths λ1 and λ2 can be located in front ofthe imager to filter out light of other wavelengths (e.g., ambientlight).

The imager can be configured to distinguish light of the two wavelengthsused for illumination. This can be done, for example, by locating acheckerboard filter pattern in front of the imager. An example of ahybrid filter for an imager that includes a bulk filter with two peaksand a checkerboard filter pattern is described in U.S. patentapplications both entitled “Method and system for wavelength-dependentimaging and detection using a hybrid filter,” application Ser. No.10/739,831, filed on Dec. 18, 2003 and application Ser. No. 10/843,517,filed on May 10, 2004, both of which are assigned to the assignee of thecurrent invention and incorporated by reference herein. FIG. 19A depictsa top view of imager 108 that includes a checkerboard filter pattern. Inthe embodiment of FIG. 19A, some imager pixels are covered by filters111 (represented by the light squares) that pass light at wavelength λ1and the other imager pixels are covered by filters 113 (represented bythe dark squares) that pass light at wavelength λ2. FIG. 19B depicts aside view of the imager from FIG. 19A with a hybrid filter 115 locatedin front of the imager. The hybrid filter includes a bulk filter 117 andthe wavelength-selective filters 111, 113. In another embodiment, onlyone set of squares (e.g., either the light or dark squares) includes awavelength-selective filter. Using a hybrid filter, the imager cansimultaneously generate the two sets of image data while theretroreflector is illuminated by reflected light from both lightsources.

Alternatively, the two sets of image data can be collected sequentially.For example, the imager and light sources can be coordinated so a firstset of image data is collected while only the first light source isactivated and the second set of image data is collected while only thesecond light source is activated. The sequential activation of lightsources and collection of the image data can be controlled by theprocessor. Although one example of sequentially collecting the two setsof image data is described, other sequential collection techniques arepossible.

Although the two sets of image data essentially cancel each other out inall areas except at the retroreflector, there are usually smalldifferences in light intensity at the non-retroreflector data points.These intensity differences are insignificant in comparison to theintensity differences at the data points that correspond to theretroreflector. In an embodiment, difference processing includesutilizing a threshold function to eliminate insignificant data points.FIG. 6 is an exemplary graph of difference image data with the intensitydifference values on the x-axis and the number of data points collectedat each intensity difference value on the y-axis. Note that the datapoints can be corrected to deal with negative intensity differencevalues. As represented by the peak 144 in the graph, most of the datapoints are clustered together. These data points represent the areaswhere the two sets of image data are roughly equivalent and essentially,but not completely, cancel each other out. There is another smallercluster of data points 146 located where the intensity difference valuesare greater. This smaller cluster of data points corresponds to thelight that is reflected from the retroreflector. As described above, theintensity difference values at these data points are larger because ofthe on-axis versus off-axis effect.

In order to easily separate the insignificant intensity differencevalues (i.e., the difference values that do not correspond to theretroreflector) from the significant intensity difference values (i.e.,the difference values that do correspond to the retroreflector), athreshold difference value is established below which intensitydifference values are considered insignificant and above which intensitydifference values are considered significant. As represented by thedashed line 148 in FIG. 6, the threshold difference value is establishedat an intensity between the two clusters of data points 144, 146. In anembodiment, the threshold difference value is pre-established or setfrom characteristics of current or prior frames and is used to simplifythe difference processing. In particular, the processor can beconfigured to discard all intensity difference values that are below thethreshold difference value. Some artifacts 147 may remain after all theintensity difference values below the threshold difference value arediscarded. Discarding insignificant data points reduces the memoryrequirements of the processor and enables faster processing and lesslatency. In one embodiment, the image processing time is reduced byprocessing only portions of an image that are near to where theretroreflector was previously found.

Intensity difference values that are determined to be significant areprocessed to identify the retroreflector. For example, all of thesignificant intensity difference values are processed to identify thecenter of the retroreflector. The location of the retroreflector'scenter can then be forwarded to a computer system through, for example,a universal serial bus (USB) interface. The processor may performadditional processing including, for example, confirming that thefeatures of the retroreflector in the difference image match the knownretroreflector characteristics and confirming that a pattern ofpotential retroreflector positions in the difference image conforms to aknown pattern of positions. For example, the difference image may becompared to known image information to determine if features in thedifference image match features of the retroreflector orretroreflectors.

Characteristics of the lighting can be established to provide good imagedata. FIG. 7 depicts an alternative embodiment of the image collectionsystem 104A from FIG. 1 in which the image collection system 104Bincludes one imager 108, multiple on-axis light sources 110, andmultiple off-axis light sources 112. In this embodiment, light sources110 generate light at wavelength λ1 and are located closer to the imagerthan light sources 112, which generate light at wavelength λ2. Themultiple light sources are used to improve the field of illumination ofthe image collection system over the system of FIG. 1. The on-axis andoff-axis image data are collected and processed as described above withreference to FIG. 1. Although two examples of light sourceconfigurations are described with reference to FIGS. 1 and 7, otherconfigurations can be used.

Although one exemplary image collection system 104A is described withreference to FIG. 1, many alternative embodiments of the imagecollection system are possible. All embodiments of the image collectionsystem collect at least two sets of image data, where one set of imagedata includes a stronger indication of the retroreflector 102 than theother set of image data. As shown below, two sets of image data can becollected in many different ways.

FIG. 8 depicts an embodiment of an image collection system 104C thatincludes two imagers 108A, 108B and a single light source 110. Thesingle light source provides light at wavelength λ1 and is locatedcloser to imager 108A than to imager 108B such that the differencebetween the illumination angle and the detection angel at imager 108A issmaller than the difference between the illumination angle and thedetection angle at imager 108B. Because of the location of the lightsource relative to the two imagers, on-axis image data is collected atimager 108A and off-axis image data is collected at imager 108B. Oncethe on-axis and off-axis image data is collected, it is processed asdescribed above to generate position information.

FIG. 9 depicts an alternative embodiment of the image collection system104D that includes two imagers 108A, 108B and light sources 152, 154that provide light of two different wavelengths. In particular, twolight sources 152 that generate light at wavelength λ1 are locatednearer to imager 108A than to imager 108B and two light sources 154 thatgenerate light at wavelength λ2 are located nearer to imager 108B thanto imager 108A. With respect to light of wavelength λ1, on-axis imagedata is collected at imager 108A and off-axis image data is collected atimager 108B. With respect to light of wavelength λ2, on-axis image datais collected at imager 108B and off-axis image data is collected atimager 108A. The configuration of FIG. 9 enables two different sets ofon-axis image data and two different sets of off-axis image data to becollected. The two imagers and the two different sets of on-axis imagedata and off-axis image data can be used to generate positioninformation in three dimensions using stereo processing or simplifiedstereo processing techniques. In an embodiment, the image collectionsystem is configured in an epipolar geometry relative to theretroreflector. FIG. 10 is a perspective view of the image collectionsystem 104D of FIG. 9 configured in an epipolar geometry. Given theepipolar geometry, the x, y, and z coordinates of the retroreflector 102at position M can be calculated by first determining the positions mland m2 of the retroreflector 102 on the imagers 108A, 108B.

In another alternative embodiment, the image collection system includesa wavelength-selective filter associated with the retroreflector. Forexample, the filter may cover the retroreflector. FIG. 11 depicts anembodiment of an image collection system 104E that includes on-axis andoff-axis light sources 110, 112 and a wavelength-selective filter 158covering the retroreflector. In the embodiment of FIG. 11, thewavelength-selective filter is transmissive at wavelength λ1 andblocking at wavelength λ2. By transmitting light at wavelength λ1 andblocking light at wavelength λ2, the wavelength-selective filterprovides a higher intensity contrast at the data points corresponding tothe retroreflector.

Because a wavelength-selective filter associated with the retroreflectorcan filter one of the wavelengths of light emitted by the light sources,the two sets of image data can be generated with the light sources atthe same illumination angle. FIG. 12 depicts an exemplary embodiment ofan image collection system 104F in which the light sources 110, 112 oftwo different wavelengths have similar illumination angles. In theexample of FIG. 12, the wavelength-specific filter 158 associated withthe retroreflector 102 is highly transmissive at wavelength λ1 andblocking at wavelength λ2. A first set of image data is collected inresponse to reflected light at wavelength λ1 and a second set of imagedata is collected in response to reflected light at wavelength λ2.Because the wavelength-selective filter associated with theretroreflector is highly transmissive at wavelength λ1 and blocking atwavelength λ2, the first set of image data includes a strongerindication of the retroreflector than the second set of image data. Thetwo sets of image data are processed as described above to generateposition information.

In another alternative embodiment, the image collection system includesone imager and one light source. FIG. 13 depicts an embodiment of animage collection system 104G that includes one imager 108 and oneon-axis light source 110. The light source generates light atwavelength, λ1, and the imager is configured with a checkerboard filter,as described above, in which half of the checkerboard squares pass lightat wavelength λ1 and half of the squares block light at wavelength λ1.Given this configuration, a first set of image data is generated inresponse to light at wavelength 11 and a second set of image data isgenerated in response to ambient light minus light at wavelength λ1.Because the light source is on-axis, the first set of image dataincludes a stronger indication of the retroreflector than the second setof image data. In an embodiment, a wavelength-selective filter islocated between the retroreflector and the imager. Thewavelength-selective filter passes light at wavelength λ1 and blockslight at other wavelengths. As described above with reference to FIGS.11 and 12, the wavelength-selective filter provides a higher intensitycontrast at the data points corresponding to the retroreflector.

FIG. 14 depicts another embodiment of a single light source imagecollection system. The image collection system of FIG. 14 utilizes twoseparate imagers 108A and 108B to collect the two sets of image data. Inparticular, on-axis image data is collected at imager 108A and off-axisimage data is collected at imager 108B. Again, a wavelength-selectivefilter can be located between the retroreflector and the imager to passlight at wavelength λ1.

Although the reflective element is described above as a retroreflector,a reflector other than a retroreflector can be used instead of theretroreflector. FIG. 15 depicts an embodiment of a system for generatingposition information that utilizes a mirror 160 instead of aretroreflector. As illustrated in FIG. 15, the off-axis light 116 isreflected by the mirror at an angle of reflection equal to the angle ofincidence. Reflecting the off-axis light at this angle of reflectioncauses the reflected off-axis light to miss the imager 108. On the otherhand, the mirror reflects on-axis light 114 back towards the imager,thus enabling the generation of two sets of image data, where one set ofimage data includes a stronger indication of the reflector than theother set of image data. In this configuration, the mirror is orientedsuch that it reflects one of the two light beams back to the imager.Alternatively, the system can be configured such that the mirrorreflects light from light source 112 back towards the imager instead oflight from light source 110.

A retroreflector can be used in a pointing application, for example, asa mouse in a GUI-based computer. In an embodiment, the retroreflectorconstitutes part of a pointing device that can be held and or moved by auser. When used in a pointing application, it is desirable to have theability to emulate certain commands via the pointing device. Forexample, it is desirable to be able to convey conventional mousefunctions such as “clicking,” “dragging,” “scrolling,” and liftdetection.

Some conventional mouse functions can be emulated by configuring thepointing device and the retroreflector such that the retroreflector canbe shielded or revealed in response to an action by the user. In oneembodiment, the retroreflector is shielded or revealed by a mechanismthat is activated by the user. FIGS. 16A and 16B depict an embodiment ofa pointing device 170 that incorporates a mechanism for shielding orrevealing a retroreflector. The pointing device includes aretroreflector 102 (FIG. 16B), a cover 172, guides 174, a finger recess176, and a return spring 178. In FIG. 16A the retroreflector (not shown)is shielded by the cover. The user can reveal the retroreflector bymoving the cover with the movement of a finger engaged with the fingerrecess 176. FIG. 16B shows the cover after it has been moved to revealthe retroreflector.

FIGS. 17A and 17B depict an embodiment of a pointing device 180 thatincorporates a mechanism for shielding and revealing multipleretroreflectors. The pointing device includes a cover 182, multipleretroreflectors 102, and a linear-to-rotation conversion mechanism 184.The linear-to-rotation conversion mechanism includes an axle 186,pivoting joints 188, a slider 173, guides 174, and a finger recess 176.In FIG. 17A the retroreflectors are shielded by the cover. A user canreveal the retroreflectors using the linear-to-rotation conversionmechanism to rotate the cover. In FIG. 17B the cover has been rotatedusing the linear-to-rotation conversion mechanism to reveal theretroreflectors through holes in the cover.

In another embodiment, the retroreflector is shielded or revealed by acombination of the structure and orientation of the pointing device. Forexample, at least one retroreflector is located at the bottom of acavity defined in the pointing device such that the retroreflector isshielded from illuminating light by the cavity walls when the pointingdevice is at some orientations with reference to the illuminating lightand is exposed to the illuminating light when the pointing device is atother orientations. FIGS. 18A and 18B depict an embodiment of a pointingdevice 190 that includes a retroreflector 102 located at the bottom of acavity 192 of the pointing device. When the cavity is not aimed towardsthe illumination light, the retroreflector is shielded by the wall ofthe cavity and is not illuminated. When the cavity is aimed towards theilluminating light, the retroreflector is illuminated. FIG. 18A depictsthe pointing device oriented such that the illuminating light 194 is notincident on the retroreflector and FIG. 18B depicts the pointing deviceoriented such that the illuminating light is incident on theretroreflector.

Although some examples of techniques for shielding and revealing aretroreflector are described with reference to FIGS. 16A-18B, many otherconfigurations are possible. In another alternative embodiment, someconventional mouse functionality can be achieved by changing somecharacteristic of the retroreflector or retroreflectors to emulate amouse function. For example, the shape, size, and/or pattern of aretroreflector or retroreflectors can be changed to emulate a mousefunction. In another alternative embodiment, some functionality can beachieved by gradually revealing or shielding a retroreflector. Forexample, the scrolling speed can be increased as more of aretroreflector is revealed or decreased as more of the retroreflector isshielded.

The above-described techniques for generating position information canbe incorporated into a position tracking system for a GUI-basedcomputer. FIGS. 20-23 depict examples of an image collection system anda pointing device integrated with a computer system. FIG. 20 depicts aportable or “laptop” computer 200 configured with an image collectionsystem that includes two light sources 110, 112 and an imager 108mounted in the housing of the computer. The first light source 110 is anon-axis light source that generates light at wavelength λ1 and thesecond light source 112 is an off-axis light source that generates lightat wavelength λ2. The configuration of the image collection system issimilar to that of FIG. 1 with the processor 106 being included withinthe computer. A pointing device 130 that includes a retroreflector 102is used to indicate position in a GUI-based operating system. Positionof the pointing device is determined as described above.

FIG. 21 depicts a portable computer 200 configured with an imagecollection system that includes two light sources 152, 154 and twoimagers 108A, 108B mounted in the housing of the computer. Theconfiguration of the image collection system is similar to that of FIG.9. Given this configuration, stereo processing can be used to determinethe three-dimensional position of the pointing device.

FIGS. 22A and 22B depict front and side views of an embodiment of animage collection system in which an imager 108 and light sources 110 and112 are incorporated into a free standing housing 202. Referring to FIG.22A, light sources 110 emit light at wavelength λ1 and light sources 112emit light at wavelength λ2. Referring to FIG. 22B, the retroreflectoris covered with a wavelength-selective filter that is transmissive atwavelength λ1 and blocking at wavelength λ2. The operation of thisconfiguration is similar to the configuration described above withreference to FIG. 12. Specifically, a first set of image data iscollected in response to reflected light at wavelength λ1 and a secondset of image data is collected in response to reflected light atwavelength λ2. Because the wavelength-selective filter associated withthe retroreflector is transmissive at wavelength λ1 and blocking atwavelength λ2, the first set of image data includes a strongerindication of the retroreflector than the second set of image data. Thetwo sets of image data are processed as described above to generateposition information. The image collection system depicted in FIGS. 22Aand 22B can be used in conjunction with a computer. For example, theimage collection system can be connected to a computer through a wiredUSB connection or a wireless connection. In an embodiment, the head 203of the housing can be rotated to change the field of view of the lightsources and imager. For example, the head can be rotated downward to,for example, bring a desktop that is used for mouse navigation intoview.

FIG. 23 depicts a process flow diagram of a method for generatingposition information. At block 220, a reflector is illuminated. At block222, a first set of image data and a second set of image data aregenerated. The first set of image data includes a stronger indication ofthe reflector than the second set of image data. At block 224, positioninformation related to the reflector is generated using the first set ofimage data and the second set of image data.

FIG. 24 depicts a process flow diagram of another method for generatingposition information. At block 230, a pointing device that includes aretroreflector is illuminated. At block 232, on-axis image data isgenerated in response to the illumination. At block 234, off-axis imagedata is generated in response to the illumination. At block 236,position information related to the retroreflector is generated usingthe on-axis image data and the off-axis image data.

Because the image collection system does not rely on reflections from adesktop surface as is the case in conventional optical mouse devices,the position of the pointing device can be tracked even when thepointing device sits on a non-reflective surface or a microscopicallysmooth surface such as glass.

Although the system for generating position information is describedwith reference to pointing applications for GUI-based computer operatingsystems, the above-described technique for generating positioninformation can be used in other applications that involve determiningthe position of an object. One application includes, for example,tracking the motion of a body part for use in motion capture or gameplaying.

Another advantage of the above-described dual image position trackingtechnique is that the use of a reflector or a retroreflector allows theimage data to be collected with reduced exposure time compared toconventional digital photography exposure times.

Although the retroreflectors are described as being attached to apointing device, the retroreflectors could be integrated with some otherobjects. For example, the retroreflectors can be attached to clothing,to parts of the body, or any other object whose position and/ororientation is of interest.

An object in free space can be characterized in terms of six degrees offreedom, including three degrees of freedom in translation (e.g., the x,y, and z axes) and three degrees of freedom in rotation (e.g., rotationabout the x, y, and z axes). The position of an object can becharacterized in terms of, for example, the x, y, and z axes. Theorientation of an object can be characterized in terms of, for example,rotation around the x, y, and z axes.

The technique described above with reference to FIGS. 1-24 focusesprimarily on generating position information related to an object, forexample, the two-dimensional position of an object in terms of the x andy axes or the three-dimensional position of an object in terms of the x,y, and z axes. In some applications, it is desirable to determine theorientation of an object either alone or in conjunction with theposition of the object. In accordance with the invention, generatingorientation information involves capturing two sets of image data usingat least one reflector with an orientation-specific characteristic or areflector integrated into a device that includes someorientation-specific structural feature. The two sets of image data arethen used to generate orientation information. In particular, thedifference is taken between the two sets of image data and because theimage data is captured using a reflector with an orientation-specificcharacteristic or a reflector integrated into a device that includes anorientation-specific structural feature, the resulting difference imageincludes orientation-specific image information related to theorientation of the reflector.

Orientation-specific image information can be generated, for example,using a retroreflector with an orientation-specific shape, usingmultiple retroreflectors in an orientation-specific arrangement, or bylocating one or more retroreflectors at the bottom of respectivecavities of a pointing device such that the retroreflectors areilluminated only in certain orientations. Various examples of techniquesfor generating orientation-specific image information are describedbelow with reference to FIGS. 25A-36. In particular, FIGS. 25A-27Bdepict examples of a retroreflector or multiple retroreflectors thathave an orientation-specific characteristic, FIGS. 28A-32B depictexamples of a retroreflector or retroreflectors integrated into apointing device that includes some orientation-specific structuralfeature, and FIGS. 33A and 33B depict a pointing device that includes acombination of an orientation-specific pattern of retroreflectors and apattern of retroreflectors integrated into a pointing device thatincludes orientation-specific structural features.

FIGS. 25A and 25B depict an example of a retroreflector 300 that has anorientation-specific shape, in this case an arrow. Difference imagesformed in response to this retroreflector are unique at everyorientation of the retroreflector about the z axis. That is, theretroreflector has a rotationally asymmetric shape. In FIG. 25A thearrow points straight up and in FIG. 1B the arrow is rotatedapproximately forty-five degrees clockwise. The image of the arrow isidentified by capturing two sets of image data and taking the differencebetween the two sets of image data. The orientation of theretroreflector can be determined by comparing the difference image toknown image information. For example, the known image information is animage of the arrow in a known orientation. The known image and thedifference image can be matched to determine orientation. For example,the rotation needed to match the difference image to the known imagegives an indication of the orientation of the retroreflector. In theembodiment of FIGS. 25A and 25B, the retroreflector is located on thesurface of a pointing device 302.

FIGS. 26A and 26B depict an example of multiple retroreflectors havingthe same shape but configured in an orientation-specific arrangement onthe surface of a pointing device 302. In particular, the patternincludes three retroreflectors with a single retroreflector at the topof the pattern and two retroreflectors nearby each other at the bottomof the pattern. Difference images formed in response to this pattern ofretroreflectors are unique at every orientation of the arrangement aboutthe z axis. For example, in the orientation of FIG. 26A, the singleretroreflector is at the top of the pattern and the two nearbyretroreflectors are at the bottom of the pattern. In the orientation ofFIG. 26B, the pattern is rotated about the z axis by about ninetydegrees clockwise such that the single retroreflector is at the right ofthe pattern and the two nearby retroreflectors are at the left of thepattern.

FIGS. 27A and 27B depict an example of multiple retroreflectors 300 thathave different shapes in an orientation-specific arrangement. In theexample of FIGS. 27A and 27B, the retroreflectors are in the shape of asquare, a circle, a triangle, and a diamond. Difference images formed inresponse to this pattern of retroreflectors are unique in everyorientation about the z axis because the different shapes are indifferent locations in every orientation.

FIGS. 28A-28C depict an example of a retroreflector 310 that isintegrated into a device 302 that has an orientation-specific structuralfeature. In the example of FIGS. 28A-28C, the device is a pointingdevice and the orientation-specific structural feature is a cavity 304that extends into the pointing device from one of its surfaces. Thecavity is defined by cavity walls 312 and the retroreflector 310 islocated at the bottom of the cavity. FIG. 28A depicts a side view of thepointing device that identifies the cavity and the retroreflectorlocated at the bottom of the cavity. FIG. 28B depicts a side perspectiveview of the pointing device, the cavity, and the retroreflector relativeto a beam 308 of light. In FIG. 28B, the cavity and retroreflector areoriented relative to the beam of light such that the retroreflector isilluminated by the beam of light. Light that illuminates theretroreflector is reflected back in the same direction it is received.If the orientation of the cavity and the retroreflector relative to thebeam of light changes enough, then the beam of light is blocked by someportion of the pointing device such that the retroreflector is notilluminated. In orientations other than that shown in FIG. 28B, theorientation-specific structural feature occludes the retroreflector atleast in part. FIG. 28C depicts a side perspective view of the pointingdevice in which the cavity and the retroreflector are oriented relativeto the beam of light such that the retroreflector is not illuminated.Given the configuration of FIGS. 28A-28C, a stronger indication of theretroreflector will be present in the difference image when the cavityand retroreflector are aligned with the beam of light as shown in FIG.28B than in the difference image that is generated when the cavity andretroreflector are not aligned with the beam of light as shown in FIG.28C. The rotational orientation of the retroreflector and the pointingdevice about the x and y axes can be characterized based on theexistence of the retroreflector in the difference image.

In the embodiment of FIGS. 28A-28C, the alignment of the pointing deviceis a binary characterization. That is, the pointing device is determinedto be either aligned with the beam of light when a strong indication ofthe retroreflector is present in the difference image or not alignedwith the beam of light when a strong indication of the retroreflector isnot present in the difference image. In another embodiment, theorientation between the cavity, the retroreflector, and the beam oflight can be determined from the shape of the retroreflector in thedifference image or from the intensity values of data points in thedifference image that are related to the retroreflector. For example, inone embodiment, a partial image of the retroreflector can be used todetermine orientation. In another embodiment, the intensity of thedetected light from the retroreflector can be used to determineorientation. The intensity of the detected light from the retroreflectorcan change as the distance between the retroreflector and the imagerchanges and/or as orientation-specific structural features reveal orblock the retroreflector as the orientation of the pointing devicechanges. Note that orientation can be determined from the shape of aretroreflector regardless of distance by comparing the ratio of heightto width of the image.

In the example of FIGS. 28A-28C, the orientation of the pointing device302 is determined relative to rotation around the x and y axes. Asdescribed above, the retroreflector 310 disappears from the differenceimage as the pointing device is tilted to the point where theretroreflector is no longer illuminated. In some applications it isdesirable to simultaneously track the position and the orientation of apointing device. The position of the pointing device in FIGS. 28A-28Ccannot be tracked using the above-described dual image technique whenthe retroreflector is not illuminated.

In an embodiment, an additional retroreflector that is not subject tothe orientation-specific structural feature of the pointing device isplaced on a surface of the pointing device to enable position tracking.FIGS. 29A-29C depict an embodiment of a pointing device 302 that issimilar to the pointing device described with reference to FIGS. 28A-28Cexcept that it includes a retroreflector 300 that is not subject to theorientation-specific structural feature of the pointing device.Referring to FIG. 29A, the additional retroreflector is located on theouter surface 314 of the pointing device. When the pointing device isoriented as shown in FIG. 29B, both the retroreflector 300 on the outersurface of the pointing device and the retroreflector 310 in the cavity304 of the pointing device are illuminated by the beam 308 of light.Because both retroreflectors are illuminated, they will both berepresented in the difference image. When the pointing device isoriented as shown in FIG. 29C, the retroreflector 300 on the outersurface of the pointing device is illuminated but the retroreflector 310in the cavity is not illuminated because the beam of light is blocked bya structural feature of the pointing device. Even though the orientationof the pointing device in FIG. 29C prevents the retroreflector 310within the cavity from being illuminated, the retroreflector 300 on theouter surface of the pointing device is still illuminated and can beused for position tracking. The dimensions and wall angles of the cavitycan be configured to set the field of view of the retroreflector.

FIGS. 30A and 30B depict an embodiment of a pointing device 302 in whichmultiple retroreflectors 310 are used in conjunction withorientation-specific structural features 324 to enable the generation oforientation information. In the embodiment of FIGS. 30A and 30B,multiple retroreflectors 310 are located in respective cavities. Thecavities are arranged in a circle 325. Each cavity has its ownorientation-specific sensitivity and the orientation of the pointingdevice can be determined from the locations of the retroreflectors thatare represented in the difference image. Referring to FIG. 30B, theorientation-specific sensitivity of the cavities is achieved by formingcavity walls 326 and 328 at different angles. As depicted in FIG. 30B,the walls 328 of the cavities closer to the center of the circle are ata larger angle relative to a line perpendicular to the surface 314 ofthe pointing device than the walls 326 further from the center.

FIG. 31A depicts a side view of the pointing device 302 from FIGS. 30Aand 30B in which the cavities 324 and retroreflectors 310 are orientedrelative to a beam 308 of light such that all of the retroreflectors areilluminated. FIG. 31B is a front view of the pointing device from FIG.31A in which the illumination of all of the retroreflectors isrepresented by the filled in dark circles. From the illumination patternin FIG. 31B, it can be inferred that all of the cavities andretroreflectors are aligned with the beam of light. FIG. 32A depicts aside view of the pointing device from FIGS. 30A and 30B in which thecavities and retroreflectors are oriented relative to the beam 308 oflight such that only some of the retroreflectors are illuminated. FIG.32B is a front view of the pointing device from FIG. 32A in which theilluminated retroreflectors are represented by filled in dark circlesand the non-illuminated retroreflectors are represented by the un-filledin circles. The orientation of the pointing device can be determinedfrom the identity of the illuminated retroreflectors. In thisembodiment, a non-illuminated retroreflector indicates that the pointingdevice is oriented such that the non-illuminated retroreflector hasrotated towards the image collection system. Because the retroreflectorsare located on the pointing device around the perimeter of circle 325,the orientation of the pointing device relative to the x and y axes canbe determined. In some orientations, retroreflectors may be partiallyilluminated. Additional orientation information can be obtained from,for example, the shapes and/or intensity values of the image informationrelated to the partially illuminated retroreflectors.

In another embodiment, the orientation-specific structural feature is aslot that extends into a device such as a pointing device. Aretroreflector is located within the slot. Because of the shape of theslot, the field of view of the retroreflector is larger in one directionthan in the other direction. Specifically, the field of view in thedirection parallel to the slot is larger than the field of view in thedirection perpendicular to the slot. The differences in the field ofview allow the retroreflector to be illuminated over a larger range inthe parallel direction and a smaller range in the perpendiculardirection. A change in orientation in the perpendicular direction willbe indicated by the partial or complete disappearance of theretroreflector in the difference image. Characteristics of the field ofview of a retroreflector within a slot are a function of the width anddepth of the slot. The dimensions and configuration of theorientation-specific structural feature can be set to achieve thedesired response.

Multiple techniques for generating orientation information can becombined to enable the generation of more detailed orientationinformation. FIGS. 33A and 33B depict an example of a pointing device302 that utilizes an orientation-specific pattern of retroreflectors 300as described with reference to FIGS. 26A and 26B in conjunction with apattern of retroreflectors 310 that is integrated intoorientation-specific features 324 of the pointing device as describedwith reference to FIGS. 30A-32B. The configuration of FIGS. 33A and 33Benables the generation of orientation information in terms of rotationaround the x, y, and z axes. In particular, the orientation-specificpattern of retroreflectors 300 on the surface 314 of the pointing deviceenables the generation of orientation information in terms of rotationaround the z axis and the pattern of retroreflectors 310 located in thecavities of the pointing device enables the generation of orientationinformation in terms of rotation around the x and y axes.

FIG. 34 depicts an embodiment of a cavity 324 in a pointing device 302and a retroreflector 310 located at the bottom of the cavity. The cavityhas at least two walls 326 and 328 that have different angles relativeto the axis 330 perpendicular to the surface 314 of the pointing device302. In particular, one of the walls 326 of the cavity is at an anglethat is near parallel to axis 330 and the other wall 328 of the cavityis at an angle that is non-parallel to axis 330. The angles of the wallsand the depth of the cavity define the field of view of theretroreflector. Because the angles of the walls are different, theretroreflector has different fields of view at different orientations.The different fields of view are utilized as described above to give anindication of orientation. Although FIG. 34 depicts an example of acavity, other cavity configurations are possible.

Although the cavities depicted in FIGS. 30A-33B are symmetricallyoriented about a circle, the cavities could alternatively be located ina different pattern. Additionally, the field of view of the cavities canbe in a different, for example, more random configuration.

FIG. 35 depicts a process flow diagram of a method for generatingorientation information. At block 420, a reflector is illuminated. Atblock 422, a first set of image data and a second set of image data aregenerated, wherein the first set of image data includes a strongerindication of the reflector than the second set of image data. At block424, orientation information related to the reflector is generated usingthe first set of image data and the second set of image data.

FIG. 36 depicts a process flow diagram of a method for generatingposition information and orientation information. At block 430, apointing device that includes a retroreflector is illuminated. At block432, a first set of image data is generated in response to theillumination. At block 434, a second set of image data is generated inresponse to the illumination. At block 436, position information andorientation information related to the retroreflector is generated usingthe first set of image data and the second set of image data.

Although some exemplary orientation-specific shapes and patterns ofretroreflectors are described above as examples of orientation-specificcharacteristics, other orientation-specific configurations are possible.For example, retroreflectors in a line, a pattern of retroreflectorlines, or the size of retroreflectors can be used to achieve anorientation-specific characteristic. Additionally, although someexemplary orientation-specific structural features are described above,other orientation-specific structural features are possible.

Although the retroreflectors are described as being attached to apointing device, the retroreflectors could be integrated with some otherobjects. For example, the retroreflectors can be attached to clothing,to parts of the body, or any other object whose position and/ororientation is of interest.

Although specific embodiments of the invention have been described andillustrated, the invention is not to be limited to the specific forms orarrangements of parts so described and illustrated. The scope of theinvention is to be defined by the claims appended hereto and theirequivalents.

1. (canceled)
 2. (canceled)
 3. (canceled)
 4. (canceled)
 5. (canceled) 6.(canceled)
 7. (canceled)
 8. (canceled)
 9. (canceled)
 10. (canceled) 11.(canceled)
 12. (canceled)
 13. (canceled)
 14. (canceled)
 15. (canceled)16. (canceled)
 17. (canceled)
 18. (canceled)
 19. (canceled) 20.(canceled)
 21. (canceled)
 22. (canceled)
 23. (canceled)
 24. (canceled)25. (canceled)
 26. (canceled)
 27. A system for generating position andorientation information, the system comprising: a pointing devicecomprising a retroreflector; means for generating on-axis image data andoff-axis image data; and a processor configured to use the on-axis imagedata and the off-axis image data to generate position information andorientation information related to the pointing device.
 28. The systemof claim 27 wherein the retroreflector has an orientation-specificcharacteristic.
 29. The system of claim 27 further comprising multipleretroreflectors in an orientation-specific arrangement.
 30. The systemof claim 27 wherein the retroreflector is integrated with anorientation-specific feature of the pointing device.
 31. The system ofclaim 27 wherein the processor is configured to take the differencebetween the on-axis image data and off-axis image data.
 32. The systemof claim 31 wherein the processor is configured to compare a differenceimage to known image information.
 33. A system for generating positioninformation and orientation information, the system comprising: apointing device comprising a retroreflector; an image collection systemconfigured to generate a first set of image data and a second set ofimage data, the image collection system comprising a light source and animager, wherein the retroreflector, the light source, and the imager areoriented with respect to each other such that a portion of lightgenerated by the light source is reflected by the retroreflector anddetected by the imager; and a processor configured to use the first setof image data and the second set of image data to generate positioninformation and orientation information related to the pointing device.34. The system of claim 33 wherein the retroreflector has anorientation-specific characteristic.
 35. The system of claim 33 furthercomprising multiple retroreflectors in an orientation-specificarrangement.
 36. The system of claim 33 wherein the retroreflector isintegrated with an orientation-specific feature of the pointing device.37. The system of claim 33 wherein the processor is configured to findthe difference between the first set of image data and the second set ofimage data.
 38. The system of claim 33 wherein the processor isconfigured to compare the difference image to known image information.39. (canceled)
 40. (canceled)
 41. (canceled)
 42. (canceled) 43.(canceled)
 44. (canceled)
 45. (canceled)